1. Field of the Invention
The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus that generates a high-resolution image from a plurality of low-resolution images. The present invention also relates to an image processing method and an electronic appliance.
2. Description of Related Art
Recent years have seen advancements in various digital technologies, accompanied by the wide spread of image sensing apparatuses that acquire digital images through shooting using a solid-state image sensing device such as a CCD (charge-coupled device) or CMOS (complimentary metal oxide semiconductor) image sensor and display apparatuses that display such digital images. The image sensing apparatuses include, for example, digital still cameras and digital video cameras; the display apparatuses include, for example, liquid crystal displays and plasma televisions. In these image sensing apparatuses and display apparatuses, there have been proposed image processing technologies for converting a plurality of digital images shot at different times into an image of higher resolution.
One type of conventionally proposed processing for such high-resolution conversion is “super-resolution processing”, relying on estimating a single high-resolution image from a plurality of low-resolution images displaced from one another. And one conventionally proposed method for achieving super-resolution processing is the “reconstruction-based” method. In the reconstruction-based method, after a high-resolution image is estimated, this high-resolution image is degraded back to estimate the original low-resolution images from which the high-resolution image was constructed; then, based on the result of comparison between the estimated low-resolution images and the original low-resolution images, the high-resolution image is brought closer and closer to the ideal.
More specifically, in the reconstruction-based method, first, an initial high-resolution image is estimated from a plurality of low-resolution images (Step 31A); next, this high-resolution image is degraded back to estimate the original low-resolution images from which the high-resolution image was constructed (Step 32A); then the estimated low-resolution images are compared with the original low-resolution images (Step 33A) and, based on the result of the comparison, a new high-resolution image is estimated such as to make smaller the difference in pixel value at each corresponding pixel position between the two images compared (Step 34A). As the processing from Steps 32A through 34A is executed repeatedly such as to make the difference converge, the high-resolution image is brought closer and closer to the ideal.
Examples of conventionally proposed methods for achieving the reconstruction-based method include the ML (maximum-likelihood) method, the MAP (maximum a posterior) method, the POCS (projection onto convex set) method, the IBP (iterative back projection) method, etc. In the ML method, the squared errors between the pixel values of the low-resolution images estimated back from the high-resolution image and the pixel values of the actual low-resolution images are taken as an evaluation function, and a high-resolution image is generated such as to minimize the evaluation function. That is, super-resolution processing employing the ML method is based on the principle of maximum likelihood estimation.
In the MAP method, the squared errors between the pixel values of the low-resolution images estimated back from the high-resolution image and the pixel values of the actual low-resolution images are, after the probability information of the high-resolution image is added to them, taken as an evaluation function, and a high-resolution image is generated such as to minimize the evaluation function. That is, in the MAP method, an optimal high-resolution image is generated by estimating, based on “a priori” knowledge, a high-resolution image that maximizes the probability of occurrence in the “a posteriori” probability distribution.
In the POCS method, simultaneous equations are created with respect to the pixel values of the high-resolution image and the pixel values of the low-resolution images; as these simultaneous equations are solved one after another, the optimal pixel values of the high-resolution image are acquired and thereby a new high-resolution image is generated. In the IBP method, a high-resolution image is generated from a plurality of low-resolution images that are shot at different shooting positions but that overlap with one another over certain pixel positions on the subject.
In this way, to acquire a high-resolution image by super-resolution processing as described above, a series of low-resolution images is used. The series consists of a plurality of chronologically ordered low-resolution images. The super-resolution processing is performed by a super-resolution processor, which can select, out of a given series of low-resolution images, a plurality of low-resolution images that it will actually use in super-resolution processing. If the selected low-resolution images are fit for the estimation of a high-resolution image, it is possible to generate a satisfactory high-resolution image. The selection processing here is therefore important.